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Abstract 



We develop a practical approach to establish the stability, that is the recurrence in a given set, of 
a large class of controlled Markov chains. These processes arise in various areas of applied science and 
rSZ* , encompass in particular important numerical methods. We show in particular how individual Lyapunov 

functions and associated drift conditions for the parametrised family of Markov transition probabilities 
and the parameter update can be combined to form Lyapunov functions for the joint process, leading to 
the proof of the desired stability property. Of particular interest is the fact that the approach applies 
even in situations where the two components of the process present a time-scale separation, which is a 
crucial feature of practical situations. We then move on to show how such a recurrence property can 
^ . be used in the context of stochastic approximation in order to prove the convergence of the parameter 

sequence, including in the situation where the so-called stepsize is adaptively tuned. We finally show 
that the results apply to various algorithms of interest in computational statistics and cognate areas. 

iTi ! 1 Introduction: recurrence of controlled MC and compound drifts 

O 

The class of controlled Markov chain processes underpins numerous models or algorithms encountered in var- 
ious areas of engineering or science (control, EM algorithm, adaptive MCMC). Consider the space (X, £>(X)) 
where X C tt" x for some n x > 1, a parametrized family of Markov transition probabilities {Pg,9 G 0} 
(for some set O C R ne ) such that for any 9,x € Q x X, Pg(x,-) is a probability distribution on (X, £>(X)). 
The class of controlled Markov chains we consider in this paper consists of the class of processes defined 
on ((6 x X) N , (B(Q) ® B(X)) m ) initialised at some (9 ,X ) = (0,x) € 9 x X, with probability distribution 
denoted Pe^G) (and associated expectation Eg^ x (-)) and defined recursively for i > as follows, 

Xi+i\{9o, Xct, X\, . . . ,Xi) ~ Pg^Xi, ■) 

'■= 4>i+l\V0i Xq, Xi, . . . , Xi+i) , (1-1) 

for a family of mappings {tfii : O x X l+1 — > O}. The present paper is concerned with the stability of the 
sequence {9i, Xi}, or more precisely the recurrence of such a process in a set C C X x i.e. we aim to develop 
practically relevant tools to establish that {9i,Xi} visits C infinitely often Fg, x — a.s. Such a form of stability 
is central to establish important properties of the process which, depending on the context, range from the 
existence of an invariant distribution for the process or its marginals to the convergence of the parameter 
sequence {9i} to a set of values of particular interest. This is largely an open problem despite its practical 
relevance as illustrated and discussed later in the paper. The following toy example illustrates the potential 
difficulties one may face. Let X = {0, 1} and consider the transition matrix 



Po = 



l-exp(-|0|) exp(-|0|) 
exp(-|0|) l-exp(-|0|) 



with O = R. This transition matrix has ir — (1/2, 1/2) as invariant distribution and its second eigenvalue is 
A = 1 — 2exp(— \9\). Set 9i + \ = 9i + a/i[l/2 — Xi + i) for some a > 0. One could expect {9i} to converge to 
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a finite value, but following the argument of [TSl Section 6.3] one can in fact show that for some values of a, 
with positive probability {Xi} may get stuck in either states while {6i} diverges. Ergodicity is lost here due 
to the fact that Cq = x {0} or C\ = x {1} is not visited infinitely often with probability one. 

The remainder of the paper is organised as follows. In Section [5] we introduce our methodology, which 
relies on a classical Lyapunov function / drift condition to establish recurrence to a set. Of particular interest 
is that we show how individual drifts characterising the evolution from Xi to Xi + \ and 9i to can be 
combined to usefully characterise the joint dynamic, even in situation where this dynamic exhibits time- 
scale separation. In Section [3] we show how such a result can be established for a class of processes which 
covers numerous applications and in Section 2] we present generic applications of our results in the context 
of stochastic approximation [16| . In Section [S] we show how the results apply to both the AM algorithm of 
[TU] but also the coerced acceptance probability algorithm 0] El and a novel variation. 

2 Compound Lyapunov functions for some two timescale controlled 
Markov chains 

The approach we adopt throughout this paper relies on a classical Lyapunov function and drift argument 
commonly used in the (homogeneous) Markov chain setting |12| . Due to the potential time inhomogeneity 
of the process above it is useful to consider a sequence of Lyapunov functions {Wi} satisfying a sequence 
of drift conditions and leading to the following classical result, provided here together with its proof for 
completeness only. Hereafter, for any i>0we let J~i := a (9q, Xq, Xi, . . . ,Xi). 

Lemma 1. Let {Wi} be a sequence of functions Wi : 6 x X 4 [0, oo) such that for the controlled Markov 
chain defined in il.l]) for all 9,x G x X 

1. for all i > 0, ¥.0 >x [Wi{e u Xi)\ < oo, 

2. there exist C C x X, a sequence {Si, i > 1} of non-negative scalars such that 2<=i ^« = 00 ano - an 
integer i w < oo such that for all i > i w , and whenever (9i,Xi) £ C, P^— a.s. 

E e , x [W i+1 (6 i+u X i+1 ) | Fi\ < Wi{6 u Xi) - 6 l+1 . (2.1) 
Then £™ =1 Xi) eC} = oo, P e , x -a.s. 

Proof. For any k > 1 we introduce the stopping times r(fc) := inf{« > k : (0i,Xi) G C}. We proceed by 
contradiction and observe first that if the claim did not hold, then there would be an integer i w < n < oo 
such that with positive probability the stopping time r(n) would be infinite, i.e. Vg^ x (r(n) = oo) > 0. We 
establish a result similar to [T21 Proposition 11.3.3, p. 266], but take care of the inhomogeneity and do not 
require the same precision. We introduce the following notation for simplicity: Wi :— Wi(9i, X{) and for any 
to G N, T m := r(n) A m (we omit the dependence on n in order to alleviate notation). Assumption (|2.4[) 
implies that for i > n + 1 , 

Eg :X [W l+1 I{r m >i + l}] = Eg tX [W t l{T m > i + 1} + E e , x [W l+1 - Wi | T^{r m >i + l}} 

< Eg^Wtlir" 1 > i}} - E e . x [S l+1 l{T m >i + l}} . 

and consequently we can establish 

^oAEZn+i^+iHr" 1 - 1 > i}} <EeAWn+i] - E e , x [W T ™} < E e , x [W n +i} ■ 

Now, by using the trivial inequality E e , x [I{r(n) = oo}J]^„ + i<5i+iI{T m - 1 > i}] < Eg ,i[E,=„ + i^+i I { Tm ~ 
1 > i}] and the monotone convergence theorem (thanks to our assumptions on {6~i}) we obtain the contra- 
dictory statement 

oo 

Pe,x(r(n) = oo) ^ < ^[W^i] < oo . 

i=n+l 

We therefore conclude that for any i > i w , Pg. x (T(i) ~ oo) = and the result follows. □ 
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The main result of this section consists of showing that it is possible to construct joint Lyapunov func- 
tion sequences {Wi} which satisfy drifts to a set C. such that the conditions of Lemma [T] hold, from two 
separate Lyapunov functions w(9) and V(x) each satisfying an individual drift condition characterising the 
two respective updates involved in the definition of {9i,X{\ in The form of these individual drifts is 

given below in (|2.2[) and (12.3[) : it is worth pointing out that we allow the drift on w(9) to vanish with time 
since {7^} may be allowed to vanish. This is practically very relevant since in many situations of interest 
the "size" of the increments — 9i\ may vanish as i — > 00 while that of — Xj| may not. The role of 

the sequence {7^} is to accommodate the possibility of two distinct timescales for the two updates in (jl.lj) - 
examples are numerous and some will be presented later in Sections U and [5] We will consider two scenarios 
which share very similar assumptions, and will be labelled with s G {0, 1}. 

(Al) Suppose V : X — >• [l,oo) and w : 9 — > [l,oo) are two functions such that there exist functions 
A w , Ay : 9 x X — > R, a set C C 9 x X, a sequence of strictly positive integers {7,, i > 1} such that 

1. {ji} is bounded, 

2. for some integer i > 0, Pg. x — a.s. the following individual drifts hold for all i > io, 

Eg, x [w(9 i+1 ) I jy < w(6i)-j i+1 A w (& h Xi) (2.2) 
E 6iX [V(X l+1 ) I Ji] < V{Xi)-A v (ei,Xi) , (2.3) 

and Eg tX [w(Bi)] < 00 and E e , x [V(Xi)} < 00, 

3. there exist constants S £ (0, 00) and v v ,v w G (0, 1] such that 

and 

sup \A w (9,x)\ V \A v {9,x)\ < 00 . 

(e,x)£C 

The following theorem establishes two recurrence results for {9i,Xi} to C. The first result requires the 
strongest set of assumptions but also establishes a stronger result, namely that the first moment of the 
return times to C are uniformly bounded in time. The second result requires weaker assumptions but does 
not guarantee the existence of a uniform in time upper bound on characteristics of the return times. A 
particular contribution here is the rescaling of either the Lyapunov function w(9) or V(x) in order to allow 
for their respective drift terms to be compared on the same time scale. 

Theorem 1. Consider the controlled Markov chain defined in Define the sequences of functions 

{Wi : 9 x X —5- [1, 00)} and {Ui : 9 x X — > [1, 00} for i > 1 and 9,i6 9xX as follows 

Wi(e,x):=V v " (x) + w^(9)/ji and Ui(9, x) := 7i Wj(0, x) , 

where {74}, w(-), V(-), v v and v w are as in f-42P; which is assumed to hold. Then, 

1. if s — 1 and I := limsup i _ s . 00 (7~ t 1 1 — 7 2 rl ) < S, then for any Sw G (0,5 — i) there exists i\y > io such 
that for any i > iyy, whenever (9$, X,-) ^ C, ¥g tX —a.s. 

Eg, x [W i+ i(9 i+1 ,X i+1 ) I Ti] < Wi(0i,Xi)-8w , (2.5) 

and Eg tX [Wi(9i, JQ)] < 00, and J^^i I{(^t> ^i) £ C} = °°; Fg iX — almost surely 

2. if s — 0, {7^} is non-increasing then for any i > io, whenever (9i,Xi) ^ C, Wg^ x — a.s. 

Eg, x [U t+1 {9 l+1 ,X l+1 ) I Fi] < Ui{9i,Xi) - S ll+1 (2.6) 

and moreover Eg x [Ui(9i, Xi)] < 00. If in addition Xa =1 7j = oo, then Xa=i Xj) G C} = 00, 

¥g. x — almost surely. 
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Proof. We start with the scenario where s = 1. By (AdJ), Jensen's inequality and the classical concavity 
identity (1 + x) v < 1 + vx for x G [— 1, oo) and v G (0, 1], we have for any i > io and Pe jX — a.s. 

E e>a [w i+1 (^ +1> x i+1 ) i h < v^( Xl ) (i - A ^ l ;f ] Y +i- + > v -m (i-7 i+ i Aw{6i > x 



Av{di,Xi)\ _ x v m / a \fi , A u ,(6'j,Xj) 



=w A e„x,) + ( 7 - + \ - 7 -') „«-(»,) - + 



Let 5vk G (0,i5 — and «w > *o be such that sup i:>i (7 i+1 — 7, ; 1 ) < <5 — <W- Then, for all i > iw and 

{e h Xi)ic, p e , x -a. s . 

E e>:c [W i+ i(0 i+ i,X i+1 ) | 7i] < Wi^Xi) ■ 

Let C := [sup 4 >. i0 7,-( 7i + L 1 - 7* 71 )] V [sup (e:r)eC \-f 1 A w (9,x)\ V |Ay (9, x)\}. Now for any i > i and (0j, X) G 
x X we have, starting as above, 

E e . x [W l+1 (9 l+1 ,X l+1 ) | Ti] < (1 + O^V^iXf) + (1 + C0 u -[7i(7m - 7*" 1 ) + IK""^)/^ 

< (l + CfW^JQ) . 

From these inequalities we therefore deduce that for any i > i , Eg tX [Wi(9i, Xi)] < {l+C) 2ly% ~ la ^E s ^ x \Wi {9 io , X io )] < 
oo where the last inequality follows from our assumptions. For the scenario where s — with 11^(6, x) = 
JiWi(6, x) we obtain from above 

E e , x [U i+1 (e i+1 ,X i+1 ) | Fi] < U^X^ + ^+x-^WiieuXi) 

A v (6 i ,X i ) A w {6 i ,X i 



and since 

(7i+i " li)Wi{e, x) + 7l+1 ( 7< -\ - 7 r!) (0) = ( 7i+1 - 7i )V» 

which together with the fact that { 7 i} is non- increasing leads to (|2.6[) for #i,X; G C c . Notice further that 
a;) < JiWi(8, x), implying Eg )X [[/i(^i, Xj)] < oo for any i > io. We now conclude in both scenarios with 
Lemma [TJ □ 

Some comments are in order concerning the choice of the Lyapunov functions and the assumptions. First 
we clarify the role of v v and v w , which are additional degrees of freedom one may find helpful to establish 
(|2.4[) in regions of x X where Ay(9,x) (resp. A w (9,x)) is negative and of large magnitude but V (resp. 
w) is itself large. Notice also that more general concave transformations of V and w could be considered 
for the definition of W% and Ui, but we do not pursue this here. We would also like to point out that other 
Lyapunov functions of the form U°'(9, x) := 7fWj(#, x) for a > may be considered but we have found the 
scenarios a — and a — 1 to be of interest only. Finally whereas it is clear that (12. 4p is stronger for s = 1 
than s = 0, we also note that Kmsup i _ >00 ( 7i I)_ 1 — 7 j _1 ) < S — Sw implies 7» = °°- 

In the next section we consider a practically relevant scenario encountered in practice, for which we 
identify C and {Wi}, but also establish an even stronger drift than in (|2.5[) . We will show in Section [S] that 
such results are satisfied in realistic scenarios. 



3 Simultaneous (9— dependent drift conditions and stability 

The results presented in the previous section are rather abstract since neither of A w , Ay and C are specified. 
Here we add some structure, and in particular show how a simultaneous drift condition on the family of 
Markov transition probabilities Pg for 9 G O, where the dependence on 9 is explicit, can be used to prove the 
stability of the sequence {9i, Xi} to a well identified set C C x X. For ease of exposition we focus throughout 
this section on the situation where fa := fa 1i for some family of updates {fa : x X — » 0, 7 G (0,7 + ]} and 
a positive sequence {%} C (0,7 + ] N , allowing us to define the update ftj+i = fa i+1 (9i, Xi + i) for i > 0. This 
directly covers most relevant applications in computational statistics and can be easily generalized. As we 
shall see the realistic assumptions we use lead in fact to stronger results than those of the previous section. 
For any / : X — > R we use the standard notation Pgf(x) := J x Pg(x, dy)f(y). The 8— dependent simultaneous 
drift conditions we consider here are as follows. 
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(A2) The family or Markov transition probabilities {Pg,9 £ 6} is such that there exist 

1. V : X — > [1, +oo) and CcX such that sup^gc V(x) < +oo, 

2. a(-),b(-) : 6 [0, +oo) and t e [0, 1] 

such that for any S,i£6xX 

< [V(x) - a _1 (0)V l (aO] l{x £ C} + &(f)I{a; 6 C} . 

We define the level sets of a(-) (resp. w{-)): for all M > 0, A M := {6» G : a(0) < M} (resp. W M ■= 
{9 G O : < Af}) and for any set A we will denote A c the complement of A in either O or X. Notice 

that assumption (A[2]) implies that infgge a(9) > 0. The situations we are interested in are those for which 
A° M ^ for any M > 0. Hereafter it will be convenient to denote for any 6,x G x X, 7 G (0,7 + ] and 
/:6xX4 K"/ 



Pg n f(9,x):= / P e (x,dy)f(ct> 1 (6,y),y) , 
and for any it, v G R 2 we define hV«:= max{w, u} and u Av := min{u, «}. 



(A3) The family of mappings {</> 7 : x X — > 0,7 G (0,7 + ]} is such that there exists a Lyapunov function 
w:6->[l,oo) such that (with {P g , 9 G 0}, V(-), C, a(-), &(•) and 1 as in (A[2])) 

1. liniM^oo infe G ^ f w{9) = 00 and lim M -s.oo sup^^ b{6)/w{6) = 

2. there exists j3 G [0, 1] and c(-),d(-) : [0,oo) such that for all 7 G (0,7+] and 9,x G x X, 

Pe :7 w(6», a;) < w{9) - 7 w(<9)A x)l{x £C} + d{0)I{x G C}) , 

where Vp(9,x) := c(0) + V@ (x)/w(9) and where 

3. c(-),d(-) : Q — >• [0, 00) are bounded and liniM^oo su PeeW^ [ C W v ^W] = 0> 

4. A(-) : [0, 00) ->• R is such that 

(a) A(0) > and it is continuous in a neighbourhood of 0, 

(b) there exists p& G (0,l//3] such that for all M > there exists Ca.m > such that for all 

z > M 

|A(z)| < C AM x , 

5. for any e > 

a(6»)w 1 - pA (6») 

sup — — „, . < 00 , 

Me*, ^"^(z) 

where V e := {9,x : V p {x)/w{9) > e}. 

Remark 1. The conditions above may appear abstract, but are motivated by the following concrete situations: 

1. The simultaneous fixed— 9 drift conditions (J 1 ^ can be established in numerous situations of practical 
interest. Examples are given in Section [SJ where the transition probabilities share the same invariant 
distribution, but it should be pointed out that such drift conditions can also be established in situations 
of interest where each transition kernel Pg has its own invariant distribution ng; this is the case for 
example in the context of the stochastic approximation implementation of the EM algorithm in [3]. 
Other examples can be found in|16] for algorithms used in the area of digital communications, although 
the dependence on 9 is never used. 

2. Typically the function A(-) in (AO will take the form of a polynomial, as a byproduct of a tractable 
approximation of w{4> 1 {9 1 y)) in terms of w(9). For example in the situation where ■& — <j) 1 {9,y) = 
9 + r )H(9, y), which corresponds to the standard stochastic approximation framework (see Section @|, 
a Taylor expansion of «;(■#) around 9 will lead to, 

w(fi) < w(9) + 7 (H(9, y), Vu>(0)) + ~ 7 V' x \H{9, y)\ 2 

whenever w" :— sup ege V 2 u>(0) < 00. With appropriate assumptions on H(9,y) one can apply Pg to 
both sides of this inequality and hence obtain a drift condition on w(-) of the form given in (J^j. 
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3. The condition required on p& G [0, i//3] can be understood as being a tradeoff between the strength of 
the drift in (AT2J) and the strength of unfavorable updates 9 + = </> 7 (0, x + ) such that w(9 + ) 3> w(8). 

The following proposition allows one to check (A|3])-[S] easily in numerous situations. 

Proposition 1. Let a(-), w(-), /3 and l be as in (J\3\). Assume that there exists C > and *c > such that 
for any 9 G O, a(6) < Cw"{6) and G [0, t/(l + x)]. T/ien f43)-[3 holds for V(-) as in (Jf9j). 

Proof. Consider first the casepA < 1 + xand notice that l — @pa > (1 + x— Pa)P- This leads for (0,x) G V t 
to 

F'-^w -°V*-^(a,) - vv^(s)y ~ 

Now assume that j>a > 1 + x. Since id > 1 , > 1 and t > /3j»a j using the first inequality above we deduce 
that 

a(9)w 1 ^(9) <c 
V*--Pp* (x) ~ 

□ 

Hereafter for any e G (0, A(0)) we will denote 

T e := {7,7 G (0,7+] : < 7- 1 - T 1 < A(0) - e} , 
where we omit the dependence on 7+ for simplicity. 

Theorem 2. Assume that {Pe,9 G 0} and {0 7 ,7 G (0,7 + ]} satisfy (JfB) and (J%3\). Then for any e G 
(0, A(0)) there exist A* G [1, 00), S, M* G (0, +00) such that for any 7, 7 G T £ and 9, x ^ Wa/» x C, 

fWA.y + tu/7}(6», x) < A*V(x) + w{6)/j - 5[V L {x)/a{9) + w(9)} (3.1) 

Corollary 1. Let {9i,Xi} be the controlled Markov chain process as described in Eq. hl.l)) with for any 
i > 1 <Pi(9o, Xq, X\, . . . , Xi) :— (f) li (0i-i, Xi) for a family {</> 7 : O x X — > 0, 7 G (0, 7 + ]} and some real positive 
sequence {7^}. Assume further that {Pg,9 G 0} and {</> 7 , 7 G (0, 7 + ]} satisfy (J&j), (J&j) and that {7^} is 
such that 

limsup( 7 r + \- 7 r 1 )< A(0) . (3.2) 

i— >oo 

Then, with M* as in Theorem^ the set Wjvf, x C is visited infinitely often ¥g x — a.s. by JQ}. 

Proof. (Corollary [T]). Let S G (0, 1], A* > 1 and M* > be as in Theorem [5] and define the family of 
(Lyapunov) functions {Wi(9,x) := \*V(x) + w(9)/ji}. From the assumption on {7^} there exists io G N 
such that for any i > i and 9, x ^ Wm„ x C 

P e ^W t (9,x) < Wi-i(e,x) ~ S[V l (x)/a(9) + w(9)} . 

The result follows from Lemma Q] since inf^ge w(9) > 0. □ 

Remark 2. One can notice that, 

1. in the case where 7^ = c /(ci + i) a (|3.2p is satisfied for any c > and a G (0, 1), and for Co < A(0) 
when a = 1, 

2. in the case where { 7 i = 7 < 7 + } is constant, Wo(9,x) — A*V(x) + w(9) and for any 9 G 0, a(#) < 
Cw*{9) for x > then one may show that for any i > io and 9, x ^ Wj\/„ x C 

P fl)7 Wb(0,aO < W {6,x) - mf^f^) . 
Indeed, from a standard convexity inequality, for any I G (0, 1], 

^ + (l-'M-»>( V ^)'»'-'W^ (3.3) 

which, with the choice I — 1/(1 + x), leads to 

F'(x)/V(0) + w(<9) > lV L (x)/w*(9) + (1 - r)w(<9) > l/ t/(1+ ^(a;) . 
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As a result 

V L {x)/a{8) + w(8) > hy L K 1+ *\x) + w{6)] 

> 2 - 1 -'/( 1 +-) {V{x)+w{8)) l/[1+>c) 

and we conclude. This suggests the possibility to precisely characterise the return times to Wm» x C as 
this form of drift condition is known to lead to the existence of polynomial moments of return times. 

Proof. (Theorem^ Choose e G (0, A(0)) and e_ > such that for any \z\ < e_, |A(0) — A(z)\< e/2. This 
implies, 

sup (7" 1 - 7" 1 ) - inf A(z) < A(0) -e + e/2- A(0) = -e/2 . (3.4) 

7,7er e {z:\z\<e-} 

Now let Mq > be such that sup^yyc d(8) < e_ and sup egVV c c(8) < e_/2. From (J^) and (A[3J) we have 

Mq Mq 

for (9,i)£9xX and A £ (0, 00) 

Pe^iXV + w/j}(8, x)<\ [V(x) - erH^VX*)] I{a; £ C} + A6(6»)I{a; 6 C} 

+ w{9)h - w(6)A {V p (0, x)l{x i C} + rf(6»)I{x g C}) 

Note that for (6,x) G W Mo x C c , V^(M) > M _1 and therefore from (AlHjl-IIbl 

u>(d)A(t£(0,a;)) < C A ^-xMoSup^- 1 ^) + c(8)) p * x V^(x) , 
see 

and supg e yy M A(d(6)) < 00 as A(-) is bounded on compact sets and sup eee d(8) < 00. Let now 

C' a ,m„ ■= [C A M-i M ° sup( w - 1 (0) + c{6)y±] V [ sup A(d(9))} < 00 , 
: see eew MQ 

notice that i > pa/3 and recall that V > 1, then we have for 7, 7 G T e 

P e , 7 {AF + ttf/7}(0, s) < AV(s) + w(8)/j + A{8, x) , 

with 

A(8, x) := -XV{x) + A [V(x) - o -1 (0)V*(a:)] I{x <£ C} + Afo(6»)I{x G C} 

+ (A(0) - e)w(8) -A{Vf)(0,x)I{x £ C} + d(0)I{a: G C})w(0) I{0 G W c Mg } 

+ C' AtMo V l (x)I{0eVV Ma }. (3.5) 

It will be convenient below to refer to the following inequality 

A(6,x) < -S[V L (x)/a(8)+w(8)} , (3.6) 

for 9, x £ x S and various instantiations of S, M, A > and S C X. Our ultimate aim is to prove 
that under the stated assumptions there exist 8, A* G (0, +00) and M* > Mq such that (|3.6|) holds for 
(6, x) G (Wm, x C) c . For any M > Mo we use the following partition 

(W M x C) c = (W Mo x C c ) U (W c Mo x C c ) U (W r M x C) , 

which leads us to consider three cases, (a), (b) and (c) from left to right, 
(a) For (8, x) G Wm x C c and any A > we have 

A(8, x) < [A(0) - e]w{8) - XV\x)/a{6) + C'^ Mo V\x) . 

< [A(0) -e] sup w{8) + V L {x) 
eew Mo 



c 'a,m» - V SU P a W 
«ew„ 
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where we note that sup tfeVV a(d) < oo from (A[3])-[T] Now, from our choice of Mq and since V > 1 and 
inf,j e e a ($) > 0, we conclude about the existence of A a , S a > such that for all A > A a , (8, x) £ Wm x C c 



A(0,x) < [A(0) - e]M + V L {x) 



c 'a,Mo ~ V SU P °W 
■Sew M n 



< -6 a [V L (x)/a{9) + w{9)} 



Therefore (|3.6[) is satisfied with M = Mq, any A > A a and 5 
(b) For (8, x) € yVti x C c and any A > we have 

A(8,x) < -\V L (x)/a(8) + [A(0) - £ 



A(V (O,x))]w(0) 



and we seek to show that there exists A& = A^ V An and 5b = Si A Su > (where A^ , Xu > and Si , Su > are 
given below in the proof) such that for all A > Ah and (8, x) £ x C c (|3.6p is satisfied with 5 = Sb- In what 
follows we will use the following intermediate results. From (jA[3])-[5] we have that for any (9,x) £ Wm x C c 
the condition 



w{6) 



V {9,x 

and therefore that for q £ {0,pa} 



c{8) > e_, implies > e_ 



sup c(i?) > e_/2, 



sup „ 

(W M xC)n{8,x:Vi3(e,x)>e-} V 9P \ x ) 



<a < 



(3.7) 



> a{0)w{9) ( e _/ 2 ) PA 



Indeed the case g = pa is true by assumption and for V 13 (x)/w(8) > e_/2 

V<--paP(x) ~ V L (x) V u>(0) ) ~ V L {x) 

from which we conclude. We now partition W£f x C c by considering the two following subsets. 

(i) From our choice of Mq and e_ and (|3.4j) we deduce that on the subset (W£ fo x C C )D{8, x : Vp (9, x) < e_ } 

A(0)-s-A(Vp(9,x))<-e/2, 

and consequently 

A(6»,x) < -XV"{x)/a{9) - w{8)e/2 , 

and we conclude about the existence of \,8i > such that (|3.6[) holds for any A > Ai and (5 = Si. 

(ii) By our assumption on A(-) there exists C' A e _ such that for any z > e_ 

A(0) - e- A(z) < C' A t _z p * . 
Consequently we deduce that on (W£f x C c ) n {9, x : Vp(9, x) > e_} 



Ho,x)<^ 



Y2M 



a(8)w(6) 



< Y1M 

- o(0) 



A,e_ V^x) ^ W (Q) 



c{9) 



A 



We now choose A > 2 PA C Ae _C £ _, and from (|3~71) with g = 0we have V L (x)/a(9) > C~}w(9) and therefore 

A(8,x) < \ [Y^L +W (8)C;}\ (^C A , e _C e _ - A) . 

We conclude about the existence of Xu,8u > such that Q3.6[) holds for any A > An and (5 = Su . 

(c) First, we note from our choice of Mq, (|3.4p and (|3.5p that for any 8,x £ x C and A > the 

function A(8,x) is upper bounded by 

A{9, x) < -XV{x) + Xb(9) + [A(0) — e — A (d(0))] u>(0) 
< -AV(at) + [A&(0)/io(0) - e/2]u>(0) . 

We now show that for any A G (0, +oo) there exist M> e [Mq,+oo) and S\ £ (0, +oo) such that for all 
9,x £ V\?m x (|3.6p is satisfied with 5 = <$a a function of A. From our last inequality and since i £ [0, 1] 
and V > 1* for any M > M and 6», x £ W L M x C 

A(9,x) < -A[inf tfe0 a(d)] V L (x)/a(9) + [A sup b{d)/w{d) - e/2]w{9) . 



We conclude about the existence of M\ and 5a as above from our assumption on &(•). 

We now conclude by letting A* > A a ,6 = A a V Af,,M» > Ma qV a 6 and 8 — 8\ a y\ b A S a A St,. 



□ 
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4 The central role of stability in the context of stochastic approxi- 
mation with Markovian dynamic 

In this section we illustrate the central role played by the form of stability considered in this paper to establish 
that some controlled Markov chains of practical relevance possess some desired properties. We focus on a 
particular class of controlled Markov chains driven by a so-called stochastic approximation recursion (also 
known as the Robbins-Monro algorithm). The motivation for such algorithms, described below, is to find 
the roots of the function h(-) : -> I" 8 

h{9) := [ H{9,x)n g {dx) , 

for a family of functions {H(8, i):8xX^6} and a family of probability distributions {irg, 9 £ 9} defined 
on some space X x B(X). This is a ubiquitous problem in statistics, engineering and computer science. These 
roots are rarely available analytically and a way of finding them numerically consists of considering the 
following controlled Markov chain on ((6 x X) N , (S(0) ® £(X))® N ) initialised at some (9 ,X ) = (9,x) G 
x X and defined recursively for a sequence of stepsizes {7^} for i > 0, 

Xi+i\J~i ~ Pg^Xi, •) 

9 i+1 = 9 i + li+l H{9 i ,X i+1 ) (4.1) 

where {Pg,9 G 0} (for some set C R" s ) is a family of Markov transition probabilities such that for each 
9 G 0, Pg leaves Kg invariant, i.e. is such that iigPg = Tt$. The rational for this recursion is as follows. Let 
us first rewrite the Robbins-Monro recursion as 

9 l+ i = 9i + 7 i+ i [h(0i) + 

where = H(9i,Xi+i) — h(9i)}, which is traditionally refered to as the "noise". Then {6^} can be thought 

as being a noisy version of the sequence {9i] defined as 9i+i = 9% + r yi+ih(9i), and it is believable that the 
properties of {6^} are closely related to those of the noiseless sequence {9j} provided the average effect of the 
noise on this sequence is negligible. This requires some form of averaging, or ergodicity, property on {£;}. 

The convergence of such sequences has been well studied by various authors, starting with the seminal 
work of [16] . under various assumptions on all the quantities involved. A crucial step of such convergence 
analyses, however, consists of assuming that the sequence {9i} remains bounded in a compact set of 
with probability one. This problem has traditionally been either ignored or circumvented by means of 
modifications of the recursion (|4.1|) . Indeed, one of the major difficulties specific to the Markovian dynamic 
scenario is that {9i} governs the ergodicity of {X{\ (and hence and that stability properties of {9i} 

relying on those of {#.;} require good ergodicity properties which might vanish whenever {9i} approaches a 
set 90 away from the zeroes of h(9), resulting in instability. Most existing results rely on modifications of 
the updates {</> 7 } designed to ensure a form of ergodicity of {^} which in turn ensures that {9i} inherits 
the stability properties of {§i}. The only known results we are aware of where stability is established for 
(|4.ip without any modification are [THl Part II, Section 1.9], where assumption (1.9.3) may not be satisfied 
in numerous cases of interest or directly verifiable, and |14| in a particular scenario. 

The approach we follow here is significantly different from that developed in the aforementioned works 
and consists of dividing the difficult problem of proving boundedness away from 90 into two simpler tasks. 
First using the results of Sections [2] or [3] one may establish that the sequence {8i,Xi} visits some set 
W x C C x X infinitely often Pg jX -a.s, a set which has the particularity that transition probabilities 
{Pe,9 G W} have uniformly good ergodicity properties. Then, using these facts one can show that {#,} 
follows the trajectories of the deterministic recursion {di} more and more accurately at each visit of W x C, 
and eventually remains in a set only slightly larger than W provided the deterministic sequence is itself 
stable. The advantage of our approach is that instead of aiming to establish ergodicity properties of {^} 
in worse case scenarios for the sequence {9i} we decouple the analysis of the behaviour of {9i} when it 
approaches 90 from the study of the ergodicity properties of which need to be studied for "reasonable" 
values of 9 only. Before stating the main result of this section we state our assumptions. 

(A4) Let {H(9,x)}, {%}, {^e} and {Pg} be as above. We assume that 
1. there exists 7+ > such that, 
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(a) 7 :={ 7j } C [0, 7 T, 

(b) for any 9, x £ x X and 7 G [0, 7^ 



9 + -yH(6,x) G 9 , 



2. # : 9 x X -> R"« is such that for any G 6, J x a^Mete) < +00, 

3. and for any G 6, 7rePg = 7rg. 

A practical technique to prove the boundedness of the noiseless sequence consists, whenever possible, of 
determining a Lyapunov function w : O — > [0, 00) such that (Vw(6), h(9)} < away from the roots of h(9), 
where Vw denotes the gradient of w with respect to 9 and for u, v G R™, (■, ■) is their Euclidian inner product 
(we will later on also use the notation \v\ = y/ (v, v) to denote the Euclidean norm of v). Note that although 
we use here the same symbol w as in Sections [T] or [31 the Lyapunov function below might be different. 

(A5) is an open subset of R™ e , h : O — > R" 8 is continuous and there exists a continuously differentiable 
function w : Q — > [0, 00) such that, 

1. there exists Mo > such that 

C := {9 G 6, (Vw(9),h(9)) =0}c{fle9, w(9) < M } , 

2. there exists Mi G (Mo, 00] such that Wm ± is a compact set, 

3. for any 9 G 6 \ C, (Vw(9),h(9)) < 0. 

We now introduce some additional notation needed to describe the ergodicity properties of {^} every 
time the sequence {9i,X{\ visits some set W xC. More precisely, consider the stochastic processes {i)i,Xi} 
defined on ((9 x X) N , (B(Q) <g> B(X)) m ) for any I > 0, initialised with $ ,X e 9 xX and such that for 
i > which uses the stepsize sequence 7^ := {7;+;, i > 0} 

K^o > -£o >-£i> . • • ,Xi) ~ Ptf^Xi, •) 

= ^ + 7i+ i+,ff X l+1 ) . (4.2) 

In order to take the shift in the stepsize sequence into account we denote by Pj, and the associated 

probability distribution and expectation operator for i)j = ^ £ 9 and Xo = X G X, and point out that in 
contrast to ¥g x defined earlier for {9 i ,X i ] 1 the notational dependence on 7^™' for I > is here crucial. For 
any M > we define the exit time from the level set Wm> c(Wm) := inf {k > 1 : ^ Wa;} (with the 
standard convention that inf{0} = +00) and for any j > 1 we define ?j := H($j-i, Xj) — h($j-i). We adapt 
the reasoning of [3, Proposition 5.2] (see also [2]) in order to establish the following result. 

Theorem 3. Assume (JUty and (J^), that {7^} is such that lim sup^^ 7, = and tef M G (Mo, Mi]. 
Assume that there exists C C X such that 



1. for any e > 0, 



-1 



limsup sup Fj x [ sup fe > x I{<t(Wm) > 



> e < 1 , (4.3) 



2. for any S,i60xX, 

Pe,x (nr=i U£ fc {(fli, X,) G W Mo x C}) = 1 , 
i.e. {9i,Xi} defined by Eq. (I4.1[) visits Wm x C infinitely often Wg >x — a.s. 

Then the sequence {6>;} as defined by Eq. (|4.1[) is such that 

p*,* (ur=i ng, G %}) = 1 , 

that is {0i} eventually remains in WW, Pe.^— a.s. 
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Remark 3. Proving Eq. (|4.3p is now rather well understood in general scenarios as soon as some form of local 
(in 9) uniform ergodicity of {Pe} is satisfied and can be checked in practice; see [5] and [3] for example and the 
recent results in pQ. In the present paper we rather focus on finding verifiable conditions on {7^}, {H(9,x)} 
and {Pe} which ensure that {9i,Xi} as defined by Eq. (|4.1[) visits Wm x C infinitely often ¥g iX — a.s., which 
in combination with the aforementioned existing results will allow us to conclude about the stability of a 
large class of controlled MCMC algorithms. 

Proof. For M G (Mo, Mi] we let Sq > and Ao > be as in Theorem [7] from (3j Proposition 5.2] given below 
for convenience. We consider the sequence {Ti,i > 1} of successive return times to Wm X C "separated by 
at least an exit from Wm", formally defined for i > as 

T i+1 = infij > Ti + 1 : 31 G {Ti + 1, . . . ,j}/9, <£ Wm and {9 J ,X J ) G Wm x C}, 

with the conventions To = and inf{0} = +00. It will be useful below to note that for any i > 1, Tj > i. 
Let no G N be such that 7t„ < 7n < Aq. We first show that for any S,i6 9xX, 



;x (U/. 1 



} =1, 



(4.4) 



and to achieve this we establish a bound on sup e I ge x xl , 9,i (T n < +00) for n > no. Notice that from the 
strong Markov property, for any 9, x G x X and I > tiq 

Pe, x {T1+1 < +oc) = E e , x (l{T < +oo}P fll ^ (T x < +00)) . 

In addition, for any 9,i£6xX we have 

I{Tt < +oo}F eTv x Ti (Ti < +00) < I{T t < +oo}p£^ (<t(Wm) < +00) P g , x 

and for any q > 

C MWm) < +00) = (UmMWm) = fc}) . 

From Theorem we deduce that for any q > no 



a.s.. 



U>iMWm) - A:} c su P I{a(W M ) > 
,fe>i 



which implies that for any I > no (and hence T > Z > no) 



I{T < + TO }P e x (Tj < +00) < sup sup Pi 

q>l 0,xEWm o xC ' 

Consequently by induction one obtains that for any n > ?io, 



supI{cr(W M ) > k} 

k>l 



nn— 1 
sup sup 



su P I{ct(>Vm) > fc} 



k>l 



Ek 



> <5 



Result (|4.4p then follows by a standard Borel-Cantelli argument under the condition of the theorem. We 
now prove that {9k} eventually remains in Wm, ^e,x— a.s.. First notice that by construction of {Ti}, 



and for any fc > 1 



U fe >i {Tfc = +00} = U fc >i {Tk-i < +00, T k = +00} , 
{T k -i < +oo,T fe = +00} = LL>fc-i {Tk-i =m,T k = +00} 



and for any m > k — 1 

{T fc _i = m, T k = +00} = {61 G W M , I > m + 1} U {0, g % , VZ > m + 1} . 

As a result 

U>i {^fe = +00} = \J m>0 {9 t G > m + 1} U Um>o{ft £ w m„, W > m + 1} 
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Now, since by assumption 

(Um>i{^ ^ Wa/ 0! V/ > m + 1}) = 
we conclude that for any O,ie0xX 

P <M (U m >oW e w m,? > m + 1}) = 1 . 

□ 

We briefly discuss here other applications of our stability results, in particular in the situation where the 
step-size sequence is held constant. Such fixed stepsize algorithms have been popular in engineering since 
they provide the algorithms with both some form of robustness and a "tracking" ability. The analysis of 
these algorithms naturally requires one to establish stability first |16] , We would like however to point out 
another important application in the context of adaptive step-size algorithms. Indeed, the choice of {ji} is 
known to have an important impact on the convergence properties of {9{\. In particular it is well known 
that if {72} vanishes too quickly in the early iterations of (|4.ip . convergence may be very slow. A natural 
way to address this problem consists of adaptively selecting the sequence of stepsizes {7^}. A strategy due 
to Kesten and further extended by Delyon and Juditsky in [7J is as follows. Given a non-increasing function 
7(-) :— > (0,oo) and sq = consider the modification (ALGUJ of 14.11 

Algorithm 1 Adaptive step-size algorithm 
. X l+1 ~p 0l (Xi,-) 

• 9 i+1 = e i + 7 ( Si )H(e i ,x i+1 ) 

. 8 i+1 = s i + I{(H(p i -. 1) X i ) ) H(0i,X i+1 ))<Q} 



Assume for brevity that the root of h(9) — 0, is unique. The rationale behind this recursion is that 
for sufficiently regular scenarios one may expect the event Xi), H(9i, Xi+i)) < 0} to occur with 

higher probability when 9i is in a neighbourhood B(9*,e) of 9* than when outside this neighbourhood. As 
a result 7, := 7(3,) decreases slowly as long as {9i} is outside this neighbourhood of 9*, and decreases much 
faster whenever {9i} aproaches 9*. Convergence to 9* requires that 7; — > or equivalently that Si — » 00 with 
probability one. This means that one should show that for any 7 G {7(0), 7(1), 7(2), . . .} the fixed stepsize 
sequence 'dj = i?JL 1 + 7-^(^7-1 j ^Q) f° r * — 1 ' s recurrent in the aforementioned neighbourhood, which is the 
essence of the proof of [7|. Our results allow one to establish that the homogeneous Markov chain {$J, X?} 
is recurrent in some set Wm x C, the first crucial step of the proof of [7J. A detailed analysis of such a result 
is beyond the scope of the present paper. 



5 Examples: some adaptive MCMC algorithms 

In this section we illustrate how the results established in Sections [3] -Q] can be straightforwardly applied to 
a variety of adaptive Markov chain Monte Carlo (MCMC) algorithms, where the aim is to automatically 
optimally choose the parameter 9 of a family of Markov chain transition probabilities {Pq,9 € 0}, defined 
on some X C K sharing a common invariant distribution with density 7r(-) with respect to the Lebesgue 
measure. More specifically, we focus here on the symmetric random walk Metropolis (SRWM) algorithm 
with transition probability defined for (8, x, A) S O x X x £>(X) as 

Pg(x,A) = / a(x, x + z)qg(z) dz + I{x € A} / (1 — a(x, x + z)) qe(z) dz . (5-1) 
J A-x Jx-x 

where for any x,y £ X 2 , a(x,y) := 1 A 7r(y)/7r(x), and {qe{ ) : 9 £ 0} is a family of symmetric increment 
probability densities with respect to the Lebesgue measure defined on Z x £>(Z) for some Z C X. Various 
choices for qe(-) are possible. 

The AM (Adaptive Metropolis) algorithm of [TO], is concerned with the situation where X = R" x for 
some n x > 1 and 9 = \p\T] G Q = M™ x x C where C C R" xXrix is the cone of symmetric positive definite 
matrices and q g (z) := det- 1/2 ((2.38 2 /n x )(r + e A M4«x„J) X q (((2.38 2 /n x )(T + ))-V*z)) for 
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q{z) — Af(z; 0, /n x xn x ) an d some cam S (0,1)- In fact other choices for q(-) are possible as long as it is 
symmetric, that is q(z) = q(—z) for all z G Z. In [5] it is shown that in some circumstances the "optimal" 
covariance matrix for the Normal-SRWM is rV , where T n is the true covariance matrix of the target distri- 
bution 7r(-), assumed here to exist. The AM algorithm of [TU] essentially implements the following algorithm 
to estimate T on the fly. Let cam > and let Xq =i£X, then for i > and with 9i := [^i|I\] here. 

Algorithm 2 AM algorithm, iteration i + 1 

• Sample A i+1 - Pe t (X i} •) 

• Update of the tuning parameter 

Mi+i = Mi + 7i+i(Ai+i - Mi) j 

r i+ i = r< + 7 i+ i((A l+ i - - m,) t - r<) . (5.2) 



It was realised in [4] that this algorithm is a particular instance of (|4.1|) where ff : 6 x X -> 6 is 

ff(0, x) := [x - /i| (at - a«) (x - - F] T , (5.3) 
and the corresponding mean field is 

h(o) = - mKmtt - m)(mtt - m) t + r^r - r] T . 

We show in Subsection 15.21 (Theorem [5]) that the stability of these recursions is a direct consequence of the 
result of Sections |3] and a result from [14], which establishes (^[2} for a class of target distribution densities 
7r(-). The result of SectiorQ] then directly applies to the AM algorithm, leading to the conclusion that \0i} 
eventually remains in a compact set with probability one. While the boundedness of {9i} has already been 
established in |14) using different arguments our results are more general in several ways. For example 
Theorem [5] shows that the AM algorithm is stable when the sequence of stepsizes {7^} is constant, which 
opens up the way for the analysis of more sophisticated and robust versions of the AM algorithm. Theorem[S] 
also shows that the AM algorithm is also stable for heavier tailed distributions than in [T3], in the situation 
where X = R, for both decreasing or constant stepsize sequences. As should be clear from our current 
analysis, a full study of the multivariate scenario is a different (and significant) research project. 

We now turn to another type of popular adaptive scheme for the SRWM. Let X = R™* and 9 = 
M. Suppose q(-) is a symmetric probability density on X and define the family of proposal distributions 
{qe(-),& G 6} as qe(z) := exp(— 6*)g(exp(— 6)z). Let a* G (0,1) be a desired mean acceptance probability 
for the SRWM. A possible increment probability density is again q$(z) — M(z,a :— exp(0)). The following 
algorithm aims to optimise 9* in order to achieve an expected acceptance rate of a* [3] and is often used as 
one of the components of more sophisticated schemes. 



Algorithm 3 Coerced acceptance probability RWM, iteration i + 1 



• Update the state Xi,Yi, with ~ qg^-) 

= Xi + Z i+ i 

li+l with probability a(Ai, l^_|_i) 



Xi+i — 
• Update the scaling parameter 



Xi otherwise 



= 0i + ^ i+ i(a(Xi,Y i+ i) - a*) 



In Subsection 15 . II we prove the stability of {9i, Xi} for a broad class of probability densities w(-), including 
a heavy tailed scenario and situations where the stepsize sequence is constant (Theorem [6]). It should be 
pointed out that in this case, in contrast with the AM algorithm scenario, we do not require a lower bound 
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on the scaling factor exp(6*), which requires establishing for both arbitrarily large and small values of 
exp(#) and leads us to proving the new result Theorem 0] (a stability result has been proved in [17|, but 
in a less general scenario). In fact the theory we have developed suggests improvements on this standard 
algorithm whose stability can be easily established thanks to the theory developed earlier in the paper. An 
example is given below: the rationale behind the algorithm is that for very poor initialisations the increments 
on the parameter are initially large, while still leading to a stable dynamic. 

Algorithm 4 Fast coerced acceptance probability RWM, iteration i + 1 

• Update the state Xj,Yi, with ~ qg i (-) 

Yi+i = Xi + Zi + i 

\Y i+ i with probability a(Xi,Y i+1 ) 
I Xi otherwise 

• Update the scaling parameter 

9 i+ i = 9 i +j i+1 (\6 i \+l)(a(X i ,Y i+1 )-a*) . 



The proofs of stability of the three algorithms above rely on common key intermediate results. In 
Subsection 15. II we establish (A[2j) for the SRWM under two different sets of assumptions on tt(-) and q(-). In 
Subsection 15.21 we establish (A[3J> for the AM algorithm and conclude with Theorem [SJ while in Subsection 
we establish (A[3]) for the coerced acceptance algorithms, and conclude with Theorem [5] 



5.1 Establishing (AUD for SRWM algorithms 
5.1.1 The superexponential 'T + £AMln x xrij scenario 

In this section we state a result which establishes that (A[2]) is satisfied for the SRWM transition probability 
on X = W 1 * under suitable conditions on ir(-) and q(-). 

(A6) The probability distribution w(-) has the following properties: 

1. it is positive on every compact set and continuously differentiable, 

2. the exists p > 1 such that 

x 

lim sup - — -p ■ Vlog7r(a;) = — oo , (5-4) 

{x:\x\>R} Fl 

3. the contours dA (x) = {y : 7r(y) = ir(x)} are asymptotically regular, i.e. for some R > 

x Vir (x) „ „. 
sup — • < , (5.5) 

{x:\x\>R}\x\ |V7r(x)| 

4. the proposal distribution density q is that of a standardized Gaussian or Student's t-distribution. 

The following theorem quantifies the way in which ergodicity of the SRWM vanishes under as some 
of its eigenvalues become large. The norm used for matrices below is \A\ — ^jTr(AA J ) and recall that here 
9 = \p\T] e 6 = R n * x C. 

Proposition 2. Let rj £ (0,1) and V(x) oc ir^^^x). Under (AEj) one can choose V > 1 and there exist 
a,b G (0, oo) and C = £> (0, R) for some R > such that for any 9, x € X X, 



P e V(x) < (1 - a/y/det(T))V(x) + bl{x E C} 

and we note that for any T G C, 

\/det(f) < n- n */ 4 \T\ n */ 2 . 
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Proof. The first statement is proved is in |14l Proposition 15], and the second statement follows from the 
standard arithmetic/geometric mean inequality applied to the eigenvalues of T 2 , 



a. 



□ 

5.1.2 Establishing (A[2]) for the AM algorithms with weak tail assumptions 

In this section we prove (J$2§ for the SRWM algorithm on X = R in the situation where no lower bound on the 
scaling parameter of the proposal distribution is assumed and under a weaker assumption on the vanishing 
rate of the tails of the target density tt(-) than in the previous subsection. More precisely, let Pg denote 
here the random-walk Metropolis kernel with symmetric proposal distribution qe(z) = exp(— 9)q{z/ exp(0)) 
for 8 E := (—00,00). For notational simplicity, in this subsection, we introduce the piece of notation 
a = cxp(9) and use P a instead of Pi g CT , Qo instead of q\ 0%a throughout and say that a 6 exp(0). We 
will also use the piece of notation £(x) := log7r(x). We require the following assumptions on tt(-) and the 
increment proposal density q a 

(A7) 1. The target distribution tt(-) on (X,B(X)) has the following properties 

(a) It has a density ir(x) with respect to the Lebesgue measure, 

(b) 7r(x) is bounded away from on any compact set of M, 

(c) £(x) is twice differentiable. We denote £'(x) := V£(x) and £"(x) := V 2 l(x), 

(d) for any M > 0, defining e x := M/\£'(x)\, 

hm sup sup — — — = U , 

R -+°° x£B°{0,R) \t\<e x \t'(X + t)\ z 



^ 'f'(x- t) 

R~^°° xeB"(0,R) |t|<e„ \£'( x ) 



lim sup sup — Ui// — = 



M'(*-t)\ 

lim SUp SUp I — — r-r — 1 1 = . 

R -^°°xeB°(0,R)0<t<e x \£[X + t)\ 

2. The tails of ~k(x) decay at a minimum rate characterised as follows : there exist 7,p G (0, 1) such 
that 

£'{x) -k~~<{x) 
lim sup - — ; — r < and lim inf > . 

R ^xeB-(Q,R) \x\ P R^ooxeB'(0,R) \l' (x)\ 

3. The increment proposal density q a {z) is of the form q a (z) = ^q[zja) for some symmetric proba- 
bility density q(z), such that supp(q) = [—1, 1] =: Z and q(-) : Z — > [q, q] for q,q € (0, 00). 

Remark 4. Consider £(x) — C — |x| Q , for |x| > Re and a > 0. Then for x > Re, £'{x) = —ax a ~ 1 and 
£"(x) = — a(a — l)x a ~ 2 and all the conditions in (AfTJ) are satisfied. 

Remark 5. The support condition on q can be removed but this requires one to control additional "tail 
integral" terms in the proofs of this section, which would add further to already long arguments. We have 
opted for this presentation for brevity and clarity since it is the terms that we handle which are both crucial 
and difficult to handle. 

The following theorem establishes (A[2]) for the scalar SRWM with V(x) oc 7r~^(x) for some 6 (0, 1) under 
©■ 

Theorem 4. Consider the SRWM targetting 7r(-) and with increment proposal density q a . Assume they 
satisfy (j&j) and let V{x) := C7r _ ' , (x) for some 77 6 (0, 1) and c S (0,oo) such that V > 1. Then for any 
1 G (0, 1) there exists R > and do G (0, 00) such that 

1. for any x £ B c (0,R), with oT^cr) := a a /(a V o^ 2 ), 
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2. there exists a constant b £ (0, oo) such that for all a, x £ exp(0) x B(0,R) we have P a V(x) < b. 

Proof. LettG (0, 1) and R > R such that M xeB c (0iR) V l - U {x)\£' (x)\ 2 > and inf xeBO(0iR) V 1 ~'-(x)/\£'(x)\ > 
0, where i?o is given in Proposition [3J The existence of R is ensured by (A[7])-[2] and the choice of V. In- 
deed, from the assumption, for x £ B c (0 7 R) we have from Lemma [5] that V(x) > Cx,2 exp(r]C^\ \x\ p ) for 
some constant C > and |^'(a:)| > CV|x| p— 1 for x > Ri for some Ci,Re > 0, and we can conclude. From 
Proposition [3] below, for x G B c (0,R) and cr < co/|^'(x)| we have 

< V(a;) -aod 2 |f (x)\ 2 V(x) 

< V(x) - a' inf \l' (xo^V^ixo) x a 2 V L (x) . 

x o £B<=(0,R) 

For \x\ > a > cq/\£'(x)\ we have 

PaV(x) <V(x)-a' Q V(x)/\a£'(x)\ 

<V(x)-a' inf ^-'(asoJ/l^fasoJk'^'fx) 

x £B<=(0,R) 

For ct > a; > R we have 

P CT V(x) < - a |x| x F(x)/cr 
< V(x) -a' R x ^(x)/cr 

Now we can use the trivial inequalities a 2 < a < 1 < u -1 (case a £ (0, 1]) and cr -1 < 1 < a < a 2 (case 
(7 G (1, oo)) which lead to the following upper bound 

P a V{x) < V(x) - aoia- 1 A a 2 )V L (x) 

The second claim follows immediately from the bound PV(x) < 2V(x) easily obtained from (|5.9p and the 
fact that sup xGB ^ R ^ V(x) < oo. □ 

Proposition 3. Consider the SRWM targetting ir(-) satisfying (J$3\). Let V(x) := cir~ v (x) for some rj £ (0, 1) 
and c such that for all x £ X, V(x) > 1. TTien there exist a' , Rq > such that for any x £ B c (0,Rq) and 
any a £ exp(O) = (0, oo), 

pv() f^'MI 2 , ifa\£'(x)\<c 

" -1 < -a'o x < 1/K(x)|, if\x\>a>c /\£'(x)\ 



V(x) 



\x\/a, if 1 > [ x | fa 



Proof. Without loss of generality we detail the situation where x > since the case x < can be straight- 
forwardly addressed by considering the density tt-(x) := tt(—x) which also satisfies (AlTJ) , and hence Lem- 
mata [3l 0] and [5] In what follows the terms Ti(a,x) for i = 1,2,3,4 are defined in Lemma [3] Choose 
R > Rpv V R^p V Rt such that for x £ B c (0,R), \x\ > cq/\£'(x)\, where Rpv,R^j,Rt and cq are as in 
Lemmata |31 |U and [5J First from Lemma[5J we have for x > cq/\£'(x)\ and any a £ exp(6), 

paA,c /\£' (x)\ pa/\x 

Ti(a,x) = ip x {z)q a {z)dz + I{a > c /\£'(x)\} / ip x (z)q a (z)dz 

JO Jc /\£'(x)\ 

ro-f\ca/\£'(x)\ r>Ot\X 

<-e^\£'(x)\ 2 / z 2 q a (z)dz - e^I{er > c /\£'(x)\} / q a {z)dz 

Jo Jc /\e>(x)\ 

< - e^q/3\£'(x)\ 2 [o- A c /\£'{x)\f/cj - e 4 ,ql{a > cq / \£' (x)\}[a Ax- c /\£' (x)\]/a , 



and therefore for a < 



Ti(a,x) < -e^q 
1 



I{a < c /\£'(x)\}a 2 \£'(x)\ 2 /3 + l{a > c /\£'(x)\} -c 3 /\£'(x)\/a + 1 - c /\£' (x)\/a 



* [cr 2 |-g'(rr)| 2 x l{a < c /\£'(x)\} + c 3 Q /\£'(x)\/a xl{i> ff > c /\£'(x)\}] 



1G 



Now from Lemma [5] for a > x > R we have 

Ti(a,x) + T 2 (a, x) < —ex x xja 
T 3 (a,x)<0 

and for a > x — T(x) we have 

T 3 (a,x) +T 4 (a,x) < -e T x (—T(x))/a 

and we conclude with Lemma [3] and by treating the case where x < in a similar fashion. □ 

Note that, as pointed out in the proof of Proposition it is sufficient to specialise most of the results of 
Lemmata [21 [21 HI and to the case x > 0. The following lemma establishes some key properties implied 
specifically by (A[7])-[2l which will also be used in Subsection 15.31 

Lemma 2. Assume that tt(-) > 0, is differentiable and satisfies (A\Tty\^ define for any 7 > 

f°° {■nix + sgn(x)z)\ 7 f^ 1 / 7r(x) \ 7 

I-y(x) := / r ) dz and J~(x) := / — ; . . . ) dz , 

Jo \ tt(x) J J Q \ir(x - sgn(x)z) J 

with sgn(a;) := x/\x\ for x ^ and for any x > 0, T(x) := inf{y £ X : n(y) — n(x)}. Then, 

1. the function T(-) has the following properties 

(a) lim^oo T(x) = -00, 

(b) there exist constants Ct,i,Ct,2 S (0, 00) such that for all |a;| > i?r 

|T(x)| V |x| < C T ,i (-log(^(x)/C T , 2 )) 1/p , 
[or equivalents tt" 1 ^) > C* T:2 cxp (c r \[\T(x)\ p V \x\ p ] 



2. and there exists a constant C 7 £ (0, 00) such that for any igX, -f 7 (^) V J 7 (x) < C^, | a; | 1 p . 
Proof. First from assumption (A[7])-[2] there exist Re, Ce > such that for all x £ -B c (0, we have 

and consequently for all x £ B c (0, Re) and z > we have 

^ X + n(x) X)Z) = exp ( sgn(a;) /* f (x + s ^ x ^ dt ) ^ ex P (-y UN + A p - M p ]) 

Consequently for any x £ B c (0,Re) we deduce that 

7r(a:) < [tt(-^) V 7r(i? £ )] exp(Q/p|^| p ) exp(-Q/p|x| p ) (5.7) 
We deduce that there exists i?i > i?^ such that 

[ir(-Re) Vir(R e )]eMCi/p\Ri\ p )eM~C e /pR p ) < inf ir(x) , 

and from ir(-) > 0, its continuity and the fact that it is monotone on both (—00, —Re] and [R\, 00) we deduce 
the first statement. Now from (|5.7p we deduce that there exists C\ > such that for x £ \R\, 00) 

7r(a?) = 7r(T(a:)) < d exp(-Q|T(z)pO 
which implies the existence of CV.i, Cx.2, > such that for any x £ X such that |x| > i?x 

|T(x)| V |x| < Cr,i (-log(7r(x)/C ra )) 1/p (5.8) 
From above we have the upper bound 

P 00 

I-y(x) < / exp(— Ce^ijv [\ x + sgn(a:)z| p — \x\ p ])dz 
Jo 

We can conclude with the result of Lemma [7] We proceed similarly with J 7 (x) by noticing that £(x) — £(x — 
sgn(x)z) — sgn(x) f ^'(m + sgn(x)i)eft < — Ce/p[\x\ p — \x — sgn(a;)z| p ] and again conclude with Lemma[71 

□ 
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We now find a convenient expression for P a V(x)/V(x) valid for sufficiently large x and all cr's. 



Lemma 3. Assume (0Q)-[7] an d for x,r],s,z € X x (0,1) x { — 1, 1} x Z define (f> x . ri . s (z) := [ir(x + sz)/-K(x)]' n 
and 

ip x (z) := {(j) x -r,-i{z) - 1) + (^,1-^,1(2:) - l) - ((f>x,i,i(z) - l) ■ 

For any x > define T(x) := inf{j/ G X : 7r(x) = 7r(y)} and Zet V(x) oc 7r _? '(x). T/ien i/iere exists Rpv > 
sitc/i that for all x > Rpv and any a G exp(0) 



V(x) 



1 = ^2T t {o-,x) 



with 
Ti(a,x) 



ip x (z)q a {z)dz 



T 2 (cr,x) =I{a > x} I [4> x ,i-n,i( z ) ~ <PxS,i(z)} q a {z)dz , 

(ff-i+T(i))A0 

[0T(x).-r h -i(z) - l] qa(z +x- T(x))dz , 

pa— {x— T(x)) 

[<f>r(x),i- v ,-i(z) - 1 + 1 - <h(x),i -i( z )] + x- T{x))dz 



Ts(a, x) = I{<7 > x) 

T A (a,x) = 1{<T > x - TO)} 
Proof. Let r\ G (0, 1) and consider 



PaV(x) = ^V(y)mm{l,^\q a (x,y)dy + V(x) 



1 — min < 1 



My) 



V{y)q a {x,y)dy + / V{y)——q a {x,y)dy + V(x) 

n(x) 



' ir(x) 
1 - 



q a (x,y)dy 



ir(x) 



q a (x,y)dy , 



where A x :— {y G K : 7r(y) > tt(x)} and R x := {y G K : 7r(y) < 7r(x)} are the regions of (almost) sure 
acceptance and possible rejection, respectively. From this expression, we obtain 



PgVjx 

V(x) 



- 1 = 



(^-i)*(*,y)dy 

/ V(y)7r(y) \ ( n(y)\ 
\V(x)w(x) J + \ n(x)J 



q a (x,y)dy 



tt{x) 



- 1 



Qa(x,y)dy 



■*{y) 

ir(x) 



My) 

ir(x) 



q a {x,y)dy. 



(5.9) 



Notice that thanks to (A[7]) and Lemma [2] lim^^oo T(x) — —00 and that for R sufficiently large, for any 
x > R we have that A x — [T(a;), x] and R x = (—00, T(x)) U (x, 00). Then with y = x + z and by taking into 
account that the support of q a {z) is included in [—a, a] we have 



PcrVjx) 

V(x) 



- 1 = 



r,-i(z) - l)q a (z)dz + / (4> x ,i-, h i(z) - l) - (4> x ,i,i(z) - l)]q<r(z)dz 



+ I{a>x-T(x)} 



v,l- v ,-l(z) - l) - (^x,!,-! (2) ~ l)]<? CT (2:)dz 
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and therefore, because x — T(x) > x, we may write 

p v(x) r aAx r 

" \ - - 1 = / tl> s (z)q„{z)dz + I{<r>x} (4> xA - nA (z) - 1) - {<t> x ,i,i(z) - l)]q„(z)dz 

v \ x ) Jq Jx 

p(x-T(x))Ao- 

+ I{cr>x} {4> x - v -i(z)-l)q cr {z)Az 

J xAa 

+ I{a > x-T{x)} f -(0 I , 1 ,_ 1 (z)-l)]g (T (z)dz 

Jx-T(x) 

and we conclude by using that 7r(T(x)) = ir(x) and the intermediate change of variable z' — T(x) — rr + z. □ 

Here we prove some properties of ip x (z) which will allow us to upper bound the term Ti(<7, x) in the case 
where a < x. 

Lemma 4. Assume and for r\ £ (0,1) let ip x (z) be as in Lemma\^ Then there exist constants 

Co ,£,/,, i?,/> > such that for all x > R^, ipx(') ■ [0, x] (— oo,0] and ip x (z) satisfies the following upper 
bounds 

h( s, v j\£'(x)\ 2 z 2 , for0<z<co/\£'(x)\ 

for C0 /nx)\< Z <x (5 ' 10) 

Proof. Note first that for s 6 { — 1, 1}, because <fi x ,ri,s(z) ■= [tt(x + sz)/ir(x)] v = cxp[r/(£(x + sz) — £{x))], 

4>'x,ri,s( z ) = r)st'(x + sz)(p Xi11iS (z) and 

4>x, n A z ) = [n 2 \£'(x + sz)\ 2 + i]£"(x + sz)]4> x ^^ s (z) . 

We now prove the desired upper bounds on ip x {z) by considering the following three cases (a) < z < 
Cq/\£'(x)\, (b) co/|^'(x)| < z < Cq/\£'(x)\ and (c) Cq/\£'(x)\ < z < x for an appropriate choice of the 
constants Co, Cq > to be determined. 

Case (a) < z < co/|^"(cc)|. We consider a first-order Taylor expansion of tpx{z) at zo = with integral 
error form and obtain 



Mz) = zr]£'(x) + z(l - n)£'(x) - z£'(x) + / W + " (*)](« - 

= / a x . v (t)(z - t)dt 
Jo 



where 



a^ft) := r, 2 \£'{x - t)\ 2 - rj£"(x - t)]4> x ^ n ^x{t) 

+ [(1 - V ) 2 \£'(x + t)\ 2 + (l~ rj)t"(x + i)]0x,i-„,i(*) - + t)| 2 + *"(a; + t)]<fe,i,i(*) . 

We seek to upperbound a x ^{t). We choose eo 6 (0, 2n{\ — 77)) and first show that for any Co G (0, eo/2) 

lim inf ^ x Az) > 1 - e /2 (5.11) 

x^ca a<z<ca/\t'(x)\ ' ' 

Indeed, for < z < co/|£'(:r)| and x large enough to ensure £'(x) < we have for some £, x ,z & [x,x + z] the 
following Taylor expansion 

£(x + z) - £{x) = £'{x)z + -z 2 £"{x + 

cl\l"{x + ^ z )\ 
> -co - - 



\£'(x)\ 2 

and with (A[7])-[T]the last term vanishes as x — >• 00 and we conclude by the assumption that — c > — e /2. 
Now choose e 1 ,e 2 ,£3 > 0. From (|5.1ip and (A0-Q] there exists R > such that for any x > R. 

in£\z\<c Q /\l>{x)\ (t>x,i,i( z ) > l-eo/2, sup w < Co/le , {x)l \£"(x+t)\/\£'(x+t)\ 2 < a, sup| t |< Co/K , (x) | \£'(x-t)\ 2 /\£'(x+ 
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t)\ 2 < 1 + e 2 and supi t i< c /i£, (a .)| \£"(x + 1)\/\£' {x)\ 2 < e 3 /c . With these, and observing that < 0.,.,.(f) < 1, 
we obtain the following upper bound 



a x ,r,(t) < \£'{x)\- 



\£'(x + t)f 
\£'(x)\ 2 



[ V (v + ci)(l + e 2 ) + (1 - - r) + e 1 ) - (1 - Cl )(l - e /2)] 



We consider then the case where eo, eiand £3 are chosen small enough so that the term in brackets in the 
last display is negative. We note now that since for some £ Xj t G [x, x + t] 



£'(x + t) =t'(x)+U"(x + Z x>i ) 



then with < t < cq/\£'(x)\ 2 we have 



£'(x + t) \£"(x + £ x . t )\ 

> 1 - cq- 



£'{x) 



\£'(xW 



which leads to the following upperbound 



a x ,r,(t) < \£'(x)\ 2 (l - e 3 ) [v(v + d)(l + ^2) + (1 - - »? + e x ) - (1 - e x )(l - e /2)] 

Notice that by our choice of eo above, we have rj 2 + (1 — ?y) 2 — (1 — eo/2) < — — 77). Now since ei, £3 > 
can be chosen arbitrarily small above we conclude about the existence of M > 0, Co > and R > such 
that for any x > R 

sup a K < -Af|f (a;) | 2 , 

|t|<c /|«'(x)| 

and we therefore conclude that in such a case, for < z < cq/\£'(x)\ 

Mz)<-m\z 2 \£'{x)\ 2 . 

Case (b) cq/\£'(x)\ < z < Cq/\£'{x)\ . First notice that ^(O) = and inspect the derivative of this function 
and aim to prove that it is negative. For any x G X we have 

4>' x {z) = t]£'(x - z)<t> x - v ,-i{z) + (1 - r))£'(x + z)<j> Xi i- v ,i(z) - £'(x + z)<j> Xi i,i(z) 
£'{x - z) 



£'(x + z) 



£'{x + z) 



b x - v ,-i{z) + (1 - i])(l)x,i^ n ,i{z) - 4>x,i,i{z) 



Because £'(x + z) < and the two first terms in brackets form a convex combination, the second line of 
(|5.10p will be establish for co/\£'(x)\ < z < Co/\£'(x)\ once we will have shown that for x > sufficiently 
large, 

'£'(x + z) \ 

i> x -n-i{z) A (/) xA - v .i{z) . 



£'{x - z) 



Clearly (f> X} i- v ,i (2) = ^iM^) < X) i,i < 1 , so we are left with showing that 4> x ,i.i{z) < ^( x+ *j <j) x , -17,-1 (2) 



i'(x-z) 



or equivalently 

7T 

7r(x) \ ir(x) 
We consider the following Taylor expansion 



n(x + z) f ir(x — z)\ v £'(x — z) 
~ £'{x + z) 



1 



£{x + z)- £{x) + rj [£[x -z)- £(x)} = z£'{x) + -z 2 £"[x + £ X)Z ) + rj 



- z g{x) + -z 2 l"(x + t x ^) 



= (1 - n)z£'{x) + -z 2 [£"{x + £ XtZ ) + r,l"{x + &,_,)] 

for some £ XjZ G [0,2] and £ x ,_ z G [—2,0]. For now choose any Co > Co and notice that for co/\£'(x)\ < z < 
Co/\£'(x)\ we have that 

(1 -n)zt' (x)<-c(i-ti) 

s w<? + wi + + «,-)» < ^ |f,(i+e - ) l i ; ( :;; (3:+i -- )i . 
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Let ei G (0, co(l — rj)) and choose £2 > such that exp (— co(l — 77) + £1) < 1 — £2- By (A[7])-[T]we can conclude 
by letting x be sufficiently large to ensure that for cq/\£'(x)\ < z < Cq/\£'(x)\, 

tt(x + z) ( n(x — z)\ , , , . l'(x-z) 

1 ] < ex p(-c (l-» ? ) + e 1 )<l-e 2 <^ 



7r(sc) \ 7r(x) 
Now using the result of case (a) we conclude that 

il>x{z) < ^x 



£'{x + z) 



Co 



\£'{x)\ 



M 2 



Case (c) Cq/\£'(x)\ < z < x. We have the following simple bound 



tt(x) 
tt(x — z) 



- 1 



■k(x + z) 

7r(x) 



1-,, 



(5.12) 



We inspect, for Cq/\£'(x)\ < z < x and x large enough, the following difference 

£{x + z)-£{x) = [ £'{x + t)dt 
Jo 

rC /\t'(x)\ 



< 



I 

■JO 



£'(x + t)dt 

.£'(x + t). 



< -C sup 

a<t<c /\t'(x)\ £ [xj 



and we can similarly obtain a bound on £(x) — £(x — z) < — Co su Po<t<c B /\e'(x)\ \ * e( x ) \- From (A[7J)-[T] and 
the Taylor expansion £'{x + t) = £'(x) + z£"{x + ^.t) we conclude that for Co and x sufficiently large enough 
we can ensure that the upper bound in (|5.12|) is negative. 

The proof is now concluded by choosing cq as in (a), which leads to the first line of (|5.10|) . Co as in (c) 
and R large enough to cover cases (b) and (c), which imply the second line of (|5.10|) . □ 

Now in this lemma we address the situation where a > x and require an additional assumption on the 
vanishing speed of £'{x). 

Lemma 5. Assume (J^j and let Ti(<j,x) for i — 1,...,4 be as defined in Lemma\^ Then there exist 
Ct, Rt, £t > such that for x > Rt and 



1. for a > 



2. for a > x 



Ti(er, x) + T 2 (a, x) < — e T X x/a , 
T 3 (a,x)<0 , 



3. and for a > x — T(x) 

T 3 (a,x) + T 4 (er,x) < -e T (-T(x))/a . 
Proof. We start with T\ (tr, x) + Ti (er, x) and with the notation of Lemma [2] we obtain 



Ti(ct, x) + T 2 (a, x) < 



7r(x — Z) 

tt(x) 



q<j(z)dz 



Tt(x + Z) 

ir(x) 



1-17 



q a (z)dz 



< = [JJx) -x] + -h-Jx) 
a a 



For tr > x, because (f>T( x ) i—ri,i( x ) ^ 1 m the integration domain, 

T 3 (a,x)<0 . 
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For a > x — T(x) > x we have on the one hand 



,o 

T 3 (a,x)= / [(h(x) ,-r,,-i( z ) ~ Qa(z + x- T(x))dz 
Jt{x) 

< = ( T(x) + / (f)Tfx)- v ,-i(z)dz J 

o- v Jtm J 

q ( r-T(x) \ 

< = \T(x)+J <f> T{x) _ vA (z)dz I , 

< i (T(x) + G|T(x)| 1 - p ) , 
where we have used Lemma [2] On the other hand we also have 

1-17 



f (7— (x — T(x)) 

T 4 (a,x) 



7r(T(x) -z)Y ' 7T(T(x)-2) 



7t(T(x)) 7 tt(T(x)) 



+ a; — Y(x))dz , 



< gr(«™-*r ] dz 



1-17 

o V T(T(a;)) 
< £ ( _ T(a;)) i-P , 

where we have used Lemma [5] again. We now conclude. □ 
5.2 Stability of the AM algorithms 

Thanks to Theorem [T] and its corollary we know that recurrence is ensured as soon as (A[2]) and (A[3]) are 
satisfied. In the previous section we have established conditions on ir(-) and qe(-) under which (A[2]) is satisfied 
for the transition probabilities underpinning (ALG(2]). We therefore focus on checking that (A[3]) is satisfied. 
First we start with a result which together with Theorem [3] leads to the same conclusions as [14] when {7^} 
is not constant, but also to the additional stability of the time-homogeneous Markov chain {9i,Xi} when 
7i = 7o I0r an Y i > 0. 

Theorem 5. Consider the controlled MC defined by (ALC@j) for X = R n * with n x > 1 (resp. (AL(Jjty for 
X = M ), assume that 7r(-) and q(-) satisfy (J%f$ (resp. and that {7,} is non-increasing and such that 

limsup^oo 7Z j — 7 i " 1 < 1. Then for any e > there exist M, R > such that with Wm '■= G 6 : w{9) < 
M} for w(9) = 1 + \fi\ 2+e + |r| 

Pe.x (n£ U<>* {(6i,Xi) G W M x B(0,R)}) = 1 . 

The proofs of the theorem for the two sets of assumptions rely on the following proposition, which 
establishes (A|3]) for a suitable Lyapunov function w(-). 

Proposition 4. Let e > and define w : — > [0, 00) as 

w{0) :=l + | M | 2+£ + |r| 

and assume that (AWj) holds for some V : X — > [1, 00) suc/i that for some (3 G (0, 1) we have V s (x) > l + |x| 2+e 
/or a/Z x G X. Let 7 + € (0, 1). Then there exists C > suc/i t/iat /or any 7 £ (0, 7 + ], and any 8, x G x X 

w/iere &(•) : 9 -> [0, 00) is as in (4%) and A(-) : [0, 00) ->• K 

A(z) := 1 - C\z + z 1 ^ 2 ^] . 

Proof. For(x+,/^,r) e X x R" x x C and 7 G [0,1], let /x+ := /i+7[x + -/i] and T + := r+7[(^-x+)(^-x + ) T -r]. 
We have the two trivial inequalities 

l/x+l = |/i + 7[x + -/x]| < (1 -7)|/i| +7|x+| 

|r+| = |r + 7 [( M - x+)(/x - x+) T -r]| < (1 - 7 )|r| + 7 |(m - - ^+) T I 
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which imply that with w(9) := 1 + + |r|, denoting 7 := 7/(1 — 7) < 1/(1 — 7 + ), 



7 \x+\ 



1-7 N 



- w(0) < -|mI 2+£ + 7 [-|r| + |/i - .x+l 2 ] + (1 - 7 ) 2+e IH 2+e 

< 7 [-|r| + |m - .x+| 2 ] + H 2+£ [(1 - 7) (1 + 7M/H) 2+e - 1 

< 7 [-to(ff) + 1 + M 2+e + 2 (|/x| 2 + |a; + | 2 )] 

+ H 2+e [-7 (1 + 7M/H) 2+e + (i + i\x+\/\»\) 2+e 1" 

By the mean value theorem 

|/i| 2+£ [(1 + 7 M/H) 2+e - l] < lM| 2+£ (2 + e)7k+l/lMl x (1 + l\x+\/\n\) 1+e 

= 7 ^±ix| M | 1 +> + |(l + 7 |x + |/| M |) 1+e 
1-7 

and since |/i| 2+e [l — (1 + 7 |a:_|_|/|/x|) +e ] < we obtain the following bound, 



2+e 



w(0+)-w(6) < yw(6) 
for some C E (0, 00) and where 



1 2|u| 2 , ., (2 + e) |u| 1+£ |x+| , , , „ ,, 

w(fc') w(#) v ' 1 — 7 w(#) 



l+e 



< 7 ty(6»)(-l + C*(6l,a; + )) 



,w(0) + u>(<9) ) + \ u;V(i+O(0) + ^1/(1+^(0) 
Now the identity (a + fe) 1+e < 2 £ (a 1+e + b 1+e ) for a, 6 > and the following equalities 

|M| x M 1/(1+e) ImI /|x+| 2 + e \ 1/(1+e)/(2+e) 



1+e 



ttfV(l+«O(0) w l/(2+e)( 6 |) ^ 

l^ji+i/Ci+e) /|x+| 2 + e \ 1/(1+e) 



;V(i+«O(0) V w {9) 



yield 



and 



| M | X |x + |Vd+e) | X+ 



1+1/(1+0 \ 1+e 



tflV(l+e)(fl) W l/(l+0(6») 



< 2 e 



1+e 
2+e\ 5+7 



w((9) 



k+l 



w>(0) 



V(2H 



\x+ 



2+e 



w(0) 



^.(H!^,„ (e) -./, + , 

w(#) w(0) 
We therefore deduce that if for any x E X, V^(a;) > 1 + |x| 2+€ then 



m x+) < u;(0)- e/(2+e) + 1± 7^ + 2 e 

< ™(0)- e /( 2 + e ) + 2^fe^ + 2 £ 
w(#) 



|2+e 



uj(0) 

YUM 

w{6) 



1/(2+0 



1/(2+0 



k+l 



2+e 



w(0) , 

10(d) 



Now by (A[5]) and Jensen's inequality we deduce that for some constant C > 1 

P 7 ,e*(M) 

'^(i)l{i^C} + ^(«)I{i£C}\ 1/(2+£) W(a:)I{a: g C} + &"(0)I{a: E C} 



< u.(6»)- e/(2+e) + C 



,[9) 



and conclude. 
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5.2.1 Multivariate case and superexponential tails: (A[6|) 

Proof of Theorem^ under Let e > 0, /3 6 (0, 1/(1 + n x /2)] and oc tt^Oe) for some rj € (0, 1) 

where the constant of proportionality is such that V /3 (x) > 1 + |x| 2+e (which is possible as ir~ v (x) > 
C\ exp(C2|a;|) for some C\,C2 > 0). From Proposition [2] there exists a', b' > and C :— 5(0, i?) for some 
R > such that for any x e X 

JW(a:) < (1 - o'/|r + cam^x^ \ nj2 )V{x) + b'l{x e C} 

< l-a"/(\e AM In x xnJ n * /2 +w nx/2 (0j)] V(x)I{x£C} + bI{xeC} , 

where 6 = 6' + sup^gc V(x). Now from Proposition [4] with 1x1(6*) := 1 + tt| 2+e + |r|, we have 

p, < i«(9) - T «,(9)A L(9)-«/<-«) + VWH^CK^UxcC} 

with A(z) := 1 — C[z + 2rV( 2 + e )] and we conclude with Proposition [T] and Theorem[5]. □ 

5.2.2 Relaxed tail conditions, univariate scenario: (A[7]) 

Now we draw the same conclusions when X = R and ir(-) now satisfies less stringent tail conditions. 

Proof of Theorem^ under (4%). Let i,r) G (0,1) and /3 G (0,t/2]. Let V(x) oc ^(x), such that V p {x) > 
l + |x| 2+e (which is possible as ir~ v (x) > C\ exp(C2|a;| p ) for some C\ , C2 > from Lemma [5]). From Theorem 
|H there exist b,R> such that with C = B(0, R) for any 9, x e 6 X X 

P e V(x) < [V(x) - ar l (6)V\x)} l{x e C} + 61{a; G C} , 

with a-^O) = oo/[(T + 6am)- 2 V (r + 6 A m)] > ao/K^L v (wm + «>(*))] and := 1 + |m| 2+£ + |r|. From 
Proposition 0] we therefore have 

P, 7 „(9, ,) < „(9) - TU ,(9) A L(9)-/<-.> + V^NCH^.C) 

V w(6») 

with A(z) := 1 — C[z + zV( 2 + e )] and we conclude with Proposition [T] and Theorem[5]. □ 
5.3 Stability of the coerced acceptance probability algorithms 

In this subsection we establish the stability of (ALGOJ) and (ALGHJ) in a univariate setting. We proceed as 
in Section 15721 and aim to apply Theorem Q] and its corollary which require (AH]) and (AS]) to be satisfied. A 
related result has been established in [17] under a more stringent condition on the decay of the tails of the 
target density, and not covering constant stepsize sequences {7^}. 

Theorem 6. Consider the controlled MC as defined by either (ALC&j) or (ALC^j for some a* G (0,1/2). 
Assume that 7r(-) and q(-) satisfy (J^j and that the stepsize sequence {7^} is such that 

(limsup7 l ^ 1 1 - 7^) + limsup7i < a* A (- - a*) . 

Let w(9) = exp(|0|) for (ALCgfy and w{6) := 1 + |6»| 2 for (ALO^. Then there exist M, R>0 such that for 
any 9,i60xX, 

P*,* (nr=o Ui>fc {(Oi,Xi) G W M ■ xB(0, R)}) = 1 , 
i.e. TPg tX — a.s. {9i,Xi} visits Wm x B(0,R) infinitely often. 

5.3.1 Proof in the standard scenario: (ALGJ3]) 

Before starting the proofs it is worth stressing on the fact that throughout 

Pg(x,y;dx' x dy') = q(x,dy')[a(x,y')5^(dx') + (1 - a(x, y')) 8 x (dx')\ 

and hence that for any x, y G X, P$(x, y;-) — Pg(x,-) and that for notational simplicity the Lyapunov 
function V(x) should be understood as being the function V(x) x 1 defined on X 2 . 
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(Proof of Theorem\Qin the case of (AL(J3\)). First notice that there exists io G N such that sup i>io 7,^ — 

7 S ~ < a* A — a*) — 7max with 7 max := sup i>ici j{. We show that (A© and (A© are satisfied and conclude 
with Theorem [5] and CorollaryfJ] for i > i . Let r), i G (0,1), /3 G (0,t/3] and define F(x) oc ^^(x), such 
that V^(x) > 1 V (—logw(x)) 1 ' p (which is possible from Lemma|l|. From Theorem|U there exist b, R > 
such that for any 0,ie6xX 

PeV(x) < [V(x) - aT^V'ix)] l{x $ C} + M{x £ C} , 

with a(0) = [exp(6>) V exp(-26»)]/a (for some a > 0) and C = B(0,R). Now with = exp(|0|) from 
Lemma [6] there exists C > such that 

owe) Or) > 1/2 - C 7-^7 for 6>< and x E X 

exp W v / _ / e xp(-0) 

W) < C + ( ' l0g ^; /PV1 < for , > and x G X. 

v ' exp(0) exp(0) 

One can apply Proposition [SJ leading to the existence of C > such that for any 9, x G O x X, 

V"(a?)" 



By,6w(6, x) < w{6) - 71/7(0) A 
with 



c(0) 



,(0) 



I{x g C} + d(0)I{x G C} 



c(0) = C-H{\6\ < 7max }(2 + a, A - a*)) , 

= sup ' ec ^ (3:) + C- 1 !!^! < 7 max}(2 + a. A (i - a*)) , 
w(p) 2 

and 

A (z) = a* A (- - a*) - 7 max - Cz . 

Notice that from our choice of i$ sup i>io 7^ — 7^ < A(0) and that we have i/f3 > 3 > pa = 1 in (.A©- 
Now since j3 < t/3 and a(0) < w 2 (8)/ao we use Proposition [T] and conclude. □ 

Proposition 5. Consider the controlled MC as defined in (ALC&j) with a* G (0, 1/2), V(x) := ctt~ v (x) and 
assume that there exist C > and [3 G [0, 1) smc/i /j/ia/j /or aH (0, x) G x X 

sgn(^)(a exp(e) (x)-a*) < —[a* A (1/2 — a*)] + CV^(x)/ exp(|0|) . 

Let 7max G (0, a* A (1/2 — a*)). TTien for any 7 G (0,7 max ] and 9,x G x X and i/ie Lyapunov function 
w : G — »■ [1, 00) defined by w(9) := exp |0| 

P 7 ,flU>(M) <w(0)- 7 «;(0)a(i{|0| <7max}C- 1 (2 + a >t A - a,)) + , 

A (z) = a* A (- — a*) — 7 max — Cz . 

Proof. For |0| > 7 and since for all x, y+ G X 2 |a(x, y + ) — a* \ < 1, one can write (with 9+ = 9 + j[a(x, y+) — 
"*]) 

w{9+) = w(9 + 7[a(x, y + ) - a*]) 

= exp(|0| +sgn(6»)7[a(x,y + ) -a*]) . 

Now since 7 < 1, from the inequality exp(u) < 1 + u + u 2 valid for \u\ < 1, one obtains 

w(6+) <w(9)(l+ 7 sgn(9)[a(x,y + )~a*}+ 7 2 ) . 

Taking the conditional expectations yields for \9\ > 7 

Pj,ew(6, x) < k;(0)(1 + 7 sgn(0) [a cxp (e) (x) - a*] + 7 2 ) . 

< w(0) - 7 w(0) ([a* A (1/2 - a*)] - CV^(x)/exp(|0|) 7max J 
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Also notice that for any 8 £ we have < \9\ + 7, whence w(6+) < w(9) exp(7) < w(6)(l + 7 + j 2 ) < 
w(9)(l + 27) whenever 7 < 1. From this inequality and the display above we deduce for all 8, x £ O x X and 

7 e (0,7 max ], 



P^ e w{9,x) <w{9)-jw(9) 



a* A (i - a.) - 7max - I{|0| < 7max}(2 + a, A (i - a,)) - CV(z)/u>(0) 



□ 



Lemma 6. Assume that tt(-) is a strictly positive, differ entiable probability density satisfying More- 
over, suppose that q a (z) := o-~ 1 q(z/a) where q : X — > [0, q] for q > and such that it has a finite first order 
moment. Then, there exists constants C_, C+ > such that for any a* £ (0, 1/2) 



a a (x) > 1/2 — C_cr /or er < 1 and i£X 

(-logTrCaQ) 1 /" VI 
Qio-t^J < G+ /or O" > 1 ana i£X. 

£7 

Remark 6. Notice from the proof that the moment condition is assumed here in order to simplify our 
statement and that more general conditions are possible. 

Proof. For any x € X let Az(x) := {z G Z : ir(x + z)/ir(x) > 1} and Rz(x) := A\{x) (where the complement 
is with respect to Z) and A(x) := x + Az(x). Without loss of generality we focus on the case x > 0. 
From Lemma [2] there exists i?i > such that for any x > Ri, Rz{x) = (—00,—x + T(x)) U (0, 00) and 
Az(x) = [—x + T(a;),0], where T(a;) is as in Lemma[51 For, x > R\ and a < l,we have the inequalities 



a a (x) = J min |l, 



Rz(x) 



ir(x + z) 

7r(x) 
it(x + z) 
ir(x) 



q a {z)dz 

q a (z)dz 



> 1 — / q<j{z)dz 

jR z (x) 

= - - I q{z)dz 

L J — oo 

I r x I U 

> - - / q(z)dz . 
z J -00 

Now with /ii < 00 the first order moment of q we notice that from Chebyshev's inequality and for x > R% 

/>oo 

/ q(z)dz < (jR^ 1 x /ii/2 

«/ x I a 

from which we deduce the first statement for a < 1 and x > R\. Now for ir > R\ and o~ > 1 

( \ **( [ ■ J1 ![(£±£)\w ^ 

a<r(a;) < — / mm <^ 1, — — ^ dz 

°" V-'^z(x)uR Z (x) I A x ) J / 

^ c (-log(^(x)/C T , 2 )) 1/p 



where we have used the results of Lemma [2] to upper bound the Lebesgue measure of Az{x) and the last two 

rOO 
IM 



integrals. We now turn to the case < x < Ri. Let M > such that f~ q(z)dz < 1/4 and er < 1 and with 



2G 



4> x {z) = ir(x + z)/tt(x) 

M*) = i + - f (iA&±4-i) q (Z)dz 

, , ir(x + az) \ , . , f°° / 7r(a; + az) . 

- 1+ jL l 1A ^r-r (z)dz+ i { 1aA ^- 1] ' !{:]<Iz - 



7r(a; + az) ^ 



tt{x) 



>l-2 q(z)dz-2Mq sup W x ( z )\ a 

JM i£B(0,Ri),2€B(0,A/) 

and we deduce the first statement of the lemma. We now consider the case < x < R± and a > 1. There 
exists (cf . the proof of Lemma [2]) Ri > such that for all x < R% 

a<r(x) < cr 1 / q{z/a)dz + a 1 / v ; g(z/g)dz + g 1 / v ' q(z/a)dz . 

J-r 2 J-00 A x ) Jb.2 n x ) 

From the proof of Lemma[21 we have the bound ir(x + z)/tt(x) < C\ exp (— Ce/p\x — \z\\ p ) and since q(z) < q 
we deduce the existence C > such that for x <R\ and cr > 1 we have a a (x) < C /a. □ 

Remark 7. Note that the restriction a* 6 (0, 1/2) is practically harmless since this covers relevant values 
according to the scaling theory of the RWM [8] . 

5.3.2 Proof for the accelerated version: (ALGj4|). 

The arguments are similar to those of Subsection 15 . 3 . ll but w(6) is here of a different form. 

(Proof of Theorem^ in the case of (AL(J^). The beginning of the proof is similar to that for (ALGJ3J) up 
to the definition of w(9). Here we choose w(9) = 1 + \9\ 2 and from Lemma [5] we have, since for x > 
exp(x) > 1 + x 2 , the existence of C > such that 

VP(x) 

for 6 < and ieX 



*ex P (0)(z) > 1/2 -C- 



a O x P (e)0) < C 1 for > and x e X 



The proof is now identical to that for (ALGJ3]) by using Proposition [51 □ 

Proposition 6. Consider (ALCJ^ with a* £ (0, 1/2), let w(9) := 1 + \9\ 2 and assume that there exists C > 
and j3 € [0, 1) such that for any 9,ie0xX 

sgn(0)(a eMe) (x) - a t ) < —[a* A (1/2 - a*)] + CV^(x)/w{0) . 

Let 7 max S (0, a* A (1/2 — a*)). TTien £/iere exists C > smc/i f/iai /or anj/ 7 £ (0, 7 max ] o-nd ^,ie0xX, 

P e>7 w(0,a;) < w(0) - jw{9)A(I{\9\ < ljC"- 1 (a* A (1/2 - a,)) + V l3 (x)/w{9)) 

with 

A(z) = 2[a* A (1/2 - a*) - 7max - C'z] . 
Proof. With 9+ = 9 + 7(1 + |0|)[a(x, j/) - a,] we have 

w(9 + ) < w(9) + 2j\6\(\9\ + 1) sgn(e)(a(x,y) - a*) + 7 2 (1 + \9\) 2 [a(x, y) - a,} 2 
< w{9) + 2~f\9\(\9\ + 1) sgn(0)(a(z, y) - a*) + 2 1 2 w{9) 

so 

Pe n w(0, x) < w{9) + 2j\6\(\e\ + 1) [-[a* A (1/2 - a,)} + CV (x) /w{9)} + 2 1 2 w{9) 

Notice that for \9\ > 1 we have |0|(1 + \9\) > 1 + \9\ 2 . Consequently for any \9\ > 1 and ieX such that 
-[a* A (1/2 - a*)] + CV l3 (x)/w(9) < 

Pe n w(9, x) < w(9) + 2jw(9) [-[a* A (1/2 - a,)} + 7 + CV*(s)/ti;(0)] 
< w{9) - 2 1W (9) [a, A (1/2 - a,) - 7 - 2C*V r/3 (x)/w(0)] . 
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Notice that for any 9 E 0, \9\(l + \9\) < (1 + |6>|) 2 < 2(1 + |6<| 2 ). For the specific case -[a* A (1/2 - a,)] + 
CV° (x)/w(9) > we therefore have 

P g „w(9, x) < w{9) + 2~f2w(9) [-[ot, A (1/2 - a,)} + j/2 + CV fj (x)/w(9)] 
< w(9) + 2>yw{9) [-[a, A (1/2 - a*)] + 7 + 2CV fs {x)/w{9)] 

and for any 9, x E x X one has 

Pe n w{9, x) < w(9) - 2-fw(e)[-2CV (x)/w(9) - 7] . 

We can now combine these intermediate results, yielding for any 9, x E 8 x X 

Po,M > x) < w{9) - 7 2 W (0) [a, A (1/2 - a,) - 7max - I{\9\ < 1} (a* A (1/2 - a,)) - 2CV (x)/w{9))] , 
and we conclude. □ 
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A Appendix 

Lemma 7. Let c, p > be constants. Then, there exists constants M — M{c,p) G (0, 00) and xq = xo(c,p) G 
(0, 00) such that 

/■OO 

exp ( - c[(x + z) p - x p ])dz < Afi 1 ^ for all x > x . 



du 



Proof. By a change of variable u = c(x + z) p , we obtain 

poo cx p roo 

/ exp(-c[(x + z) p -x p ])dz = / e- u u?- x 

JO C P JcxP 

Integration by parts yields 

roo / -1 \ /*oo 

/ e- u up- 1 du = e- cx "(cx p )p- 1 + --1 / e 

JcxP \P / JcxP 



- 2 du 



(A.l) 



Now, if p > 1, this is enough to yield the claim. Suppose then p G (0, 1). and fix a constant A G (0, 1). By 

(ED, 

p J u 



(1 — A) / e u up l du = e cx "(cx p )p 1 + I < " u >■ 

J cx v J cx p 



du 



Now, if cx p > (| — l) j, the latter integrand is negative. Setting 



xo 



we therefore have for x > xq the desired bound 



-Of 

p ) Ac 



' exp(-c[(x + z) p -x p ])dz<±^ 
cp{\ 



A) p(l - A) 

We remark that the constant A G (0, 1) can be used to optimise the values constants M and Xq. 



□ 
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B Proofs from Section 3 



We state the following result for the reader's convenience. 

Theorem 7 (see [3] for a proof). Assume (JfBj). For any M £ (Mo, Mi] there exist 5q > and Xq > such 
that, for all n > 1, aZZ z?o £ VVa/ ; sequences p = {pk} of non negative real numbers and all sequences 
{Oc} C O n of rig -dimensional vectors satisfying 

sup pk < Xq and sup 

l<fc<ra l<k<n 

we have for k = 1, . . . , n, w(i} k ) < M, where d k = fik-i + Pkh^k^i) + Pk^k- 

References 

[1] Y. F. Atchade and G. Fort, Limit Theorems for some adaptive MCMC algorithms with subge- 
ometric kernels, Bernoulli Volume 16, Number 1 (2010), 116-154, 2010. 

[2] Andrieu C. and Moulines E., "On the ergodicity properties of some adaptive MCMC algorithms," 
Annals of Applied Probability, Vol. 16, No. 3, pp. 1462-1505, 2006. 

[3] Andrieu C, Moulines E. and Priouret P., "Stability of stochastic approximation under verifiable 
conditions", SIAM Journal on Control and Optimisation, Vol. 44, Number 1, pp. 283-312, 2005. 

[4] Andrieu C. and Robert, CP, "Controlled MCMC for optimal sampling," Tech. Rep. 0125, 
Cahiers de Mathematiques du Ceremade, Universite Paris-Dauphine, 2001. 

[5] Andrieu C. and Vihola, M., "Markovian stochastic approximation with expanding projections," 
Univ. of Bristol, Tech. Report, 2011. 

[6] Y. F. Atchade and J. S. Rosenthal, "On Adaptive Markov Chain Monte Carlo Algorithms," 
Bernoulli 11, 815-828, 2005. 

[7] Delyon, B. and Juditsky, A. Accelerated stochastic approximation, SIAM Journal on Optimiza- 
tion, vol. 3, pp. 868-881, 1993. 

[8] Gelman, A., Roberts, G., and Gilks, W., "Efficient Metropolis Jumping Rules", In Bayesian 
Statistics 5. Oxford University Press, New York, 1995. 

[9] Haario, H., Saksman, E. and Tamminen, J., "Adaptive Proposal Distribution for Random Walk 
Metropolis Algorithm", Computational Statistics, vol. 14, no. 3, pp 375-395, 1999. 

[10] Haario, H., Saksman, E. and Tamminen, J., "An Adaptive Metropolis Algorithm", Bernoulli, 
vol. 7, no. 2, 2001. 

[11] Jarner, S. F. and Hansen, E. (1998), "Geometric Ergodicity of Metropolis algorithms", Stoch. 
Proc. Appl, 85, 341-161. 

[12] Meyn, S.P. and Tweedie, R.L. (1993), Markov Chains and Stochastic Stability, Springer- Verlag. 

[13] Roberts, G.O. and Rosenthal, J.S., "Coupling and Ergodicity of Adaptive MCMC," J. Appl. 
Probab., vol 44, no. 2 , pp. 458-475, 2007. 

[14] E. Saksman and M. Vihola. On the ergodicity of the adaptive Metropolis algorithm on un- 
bounded domains. Ann. Appl. Probab., 20(6):2178-2203, Dec. 2010. 

[15] L. Younes, On the convergence of Markovian stochastic algorithms with rapidly decreasing 
ergodicity rates, Stochastics and Stochastics Reports, 65, 177-228 (1999). 

[16] A. Benveniste, M. Metivier, and P. Priouret : Adaptive algorithms and stochastic approxima- 
tions. Springer Verlag, Applications of Mathematics, vol. 22, Berlin, Heidelberg, New York, 
1990 (also available in French, Masson 1987). 

[17] M. Vihola. On the stability and ergodicity of adaptive scaling Metropolis algorithms. Stochastic 
Process. Appl., 121(12):2839-2860, 2011. 



<S , 



29 



